Typically testing and validating a software application involves executing a test suite of test cases that run the software application through various execution paths. Often the test cases are designed to test functional requirements or use cases (i.e., scenarios that represent expected customer goals in running the software application). Manually running a test suite for a complex software application and then interpreting the results is a time-consuming, error-prone, and repetitive task. Consequently, portions of the testing are often automated using a test harness-any combination of software and test data that executes the test cases in the test suite and generates test reports that summarize the results of executing the test cases in an intuitive fashion.
Test suites are often run to validate specific versions of software applications, known as “builds,” either as part of the development cycle or to ensure the quality of software applications that are targeted for customer release. However, to comprehensively validate that a software application is both functionally correct and executes applicable use cases correctly may lead to test suites that include many different test cases and, consequently, unacceptably long test cycles (i.e., time required to run the test suites). Further, depending on the order in which test cases are run, detecting critical defects may be unnecessarily delayed.
In one attempt to reduce the test cycles times, the amount of test cases that are run for different purposes (e.g., internal development builds versus release builds) may be pruned and/or the test cases may be temporally ordered. Such pruning and reordering typically reflects the priorities of program managers who define the use cases and developers who understand the functional requirements and error-prone execution paths.
Although judiciously defining test case priorities may alleviate the time required to verify a build, defining the priorities is an error-prone process. For instance, if a test case exercises multiple use cases, then the overall priority of the test case may be misconstrued. Further, the priority for certain test cases may not be defined, or may be out-of-date and no longer accurately reflect the actual priority of the test cases. Manually defining priorities also leads to inconsistently defined test cases priorities, where one organization uses a different priority for one or more test cases. In general, using conventional, largely manual, approaches to defining test case priorities often leads to incorrect prioritization of test cases and, as a result, critical tests may not be executed or may be executed unacceptably late in the test cycle. Consequently, users resort to running test suites that include all available test cases, irrespective of the unreliable test case priorities, thereby increasing the overall time required to validate builds.